Building an AI Research Assistant: Large Language Models as a Versatile Tool for Digital Historians¶

©<AUTHOR or ORGANIZATION / FUNDER>. Published by De Gruyter in cooperation with the University of Luxembourg Centre for Contemporary and Digital History. This is an Open Access article distributed under the terms of the Creative Commons Attribution License CC-BY

©<AUTHOR or ORGANIZATION / FUNDER>. Published by De Gruyter in cooperation with the University of Luxembourg Centre for Contemporary and Digital History. This is an Open Access article distributed under the terms of the Creative Commons Attribution License CC-BY-NC-ND

Large language models, GPT-4, artifical intelligence, machine learning, historical methodology, optical character recognition, oral history, prompt engineering

This article explores into the potential applications and implications of large language models (LLMs) for historical research and pedagogy. The article examines the capabilities of GPT-4 and other machine learning models through case studies studying their utility in fostering greater accessibility to historical sources. GPT-4's performance is evaluated on a series of prompted tasks including data preparation, source analysis, and the ethical implications of simulated historical worldviews. GPT-4's proficiency in historical knowledge is also evaluated by using a widely recognized machine learning benchmark. A replication study demonstrates that GPT-4 exhibits expert-level performance in three distinct historical subfields. Given the rapid advances in LLMs, historians should contribute wider debates surrounding these technologies, as the unpredictable impacts of democratized AI on historical knowledge are already emerging.

In the article's hermeneutical layer, the author explores the practice of prompt engineering, or the techniques for using natural language instructions to guide a LLM's output. Prompt engineering strategies are demonstrated through the use of few-shot prompting, chain-of-thought reasoning, and prompt chaining.

Introduction¶

In 2003, Roy Rosenzweig predicted that digital historians would need to develop new techniques "to research, write, and teach in a world of unheard-of historical abundance." () Over the past two decades, historians have risen to this challenge, embracing digital mapping, network analysis, and distant reading of large text collections as part of their methodological toolkit. () Machine learning (ML) has also emerged as a promising field in computational history, offering rapid analysis of vast datasets and making significant contributions to historical research. The range of recent applications employing ML techniques is impressive: restoring lost fragments of ancient papyri (), automated transcription of medieval manuscripts (), identifying soldiers from the U.S. Civil War through facial recognition (), and analyzing the archival records of South Africa's Truth and Reconciliation Commission to spatially and semantically map the Apartheid era ().

Yet as ML techniques increasingly become a part of the digital historian's toolkit, new research frontiers are emerging in the development of "foundational models" of artificial intelligence (AI), which possess striking capacities across a range of modalities. The scope of these capabilities and their broader implications remain intensely debated () Despite being in its infancy, this technology offers a variety of methodological possibilities for historians.

However, these advances also give historians cause for concern. The proliferation of "fake history" is a clear risk. Historians typically employ ML methods for analytical purposes, using advanced pattern recognition to interpret large collections of historical data. Yet the same techniques that provide algorithms with insights into these sources can also enable the creation of synthetic data resembling genuine artifacts. While the historical record contains numerous forgeries, advances in "generative AI" may facilitate the production of convincing disinformation on an unprecedented scale and with uncanny verisimilitude. "Deepfakes" and other forms of digital fabrication could exacerbate what many already consider a broader epistemological crisis (). Given the rapid pace of these advances, it is crucial that the profession addresses the implications of this technology. Historians will have much to contribute in contextualizing the innovative and disruptive potential of these breakthroughs.

Our engagement with this technology, however, cannot be limited to critiques from the sidelines. Historians have a compelling interest in directly engaging with both the perils and potential of "generative AI." Researchers are already exploring how these foundational models interpret historical knowledge, yielding remarkable results. Nevertheless, a critical discourse has emerged, raising key questions about how these models achieve their capabilities, the limits of their performance, and their propensity to reinforce existing inequalities. There are rich opportunities for historians to contribute to these debates. The growing maturity of these applications enables increasingly democratized access to advanced ML models (). This article proposes a starting point for historical exploration of AI by examining the capacities of one of the most studied (and controversial) AI systems: the GPT series of large language models (LLMs).

The Possibilities and Perils of Generative AI¶

As historians explore the possibilities of what is sometimes called generative AI, it is important to understand how they are created and function. With this knowledge we can better asses their strengths, weaknesses, and potential impact on historical research.

Large language models are based on gargantuan datasets created through large-scale "scraping" of the internet. Machine learning scientists employ deep learning techniques to develop statistical models that are trained on these extensive datasets. With sufficient time and computational power, the LLMs begin to exhibit a range of "emergent" capabilities (). The nature of these capacities remains a matter of intense research and debate, as do the ethical and legal questions surrounding their use. However, it is clear that these models can both interpret and generate data in ways that surpass previous ML methods. Scholars studying these AI systems have labeled them "foundational models" due to their potential to enable new domains of computational analysis ().

A variety of these foundational models have become publicly accessible. Among the best-known and most studied models is the Generative Pre-trained Transformer series by OpenAI, commonly referred to by the acronyms GPT-3, GPT-4, and ChatGPT. The GPT models are trained to statistically predict the next sequence of words (tokens) for a given text. Common examples of this capacity are the autocomplete functions in text messaging and Gmail. However, the capabilities of GPT models extend far beyond these simple applications. With their predictive abilities, GPT-3 and its successors can summarize texts, perform language translation, write working computer code, and compose strikingly informative responses on a wide array of subjects (). Indeed, the remarkable versatility of LLMs is stimulating broader discussions about the potential implictions of these technologies on society at large. ().

While the GPT series from OpenAI is the best known of these foundational models, a growing number of both commercial and open-source alternatives are available for researchers and the general public. Notable commercial LLMs include Google's Bard, Anthropic's Claude, and the models offered by Cohere. Open-source LLMs encompass BLOOM, a multilingual LLM created by the Big Science Research Workshop (), EleutherAI's GPT series, Google's FLAN instruct series (), and Meta's LLaMA models (). Additionally, several notable LLMs have been studied by researchers but are not yet accessible to the broader public, such as Google's PaLM () and DeepMind's Chinchilla ().

Foundational models are also emerging in other domains, such as image synthesis. Models like CLIP () power OpenAI's DALL-E, Midjourney, and the open-source community behind Stable Diffusion. Remarkable advances are occurring in foundational models for audio and video synthesis as well. Perhaps most notably, a combination of these models has enabled the creation of multi-modal AIs capable of working across multiple domains, such as GPT-4. ()

For a survey of the development and capacities of these models as of April 2023, see: ()

While such claims elicit both excitement and alarm, any assessment of LLMs must first be tempered with humility. LLMs are often described as possessing "knowledge" and "understanding," yet direct engagement with these models can quickly reveal both their remarkable breadth and their narrow limits. Incisive critics of this technology characterize LLMs as "stochastic parrots" that excel at uncanny mimicry of human intelligence (). A form of this mimicry has proven convincing in the past. The first attribution of artificial intelligence to a computer program occurred in 1966 with a scripted chatbot named ELIZA, developed by AI pioneer Joseph Weizenbaum (, 289-298). A recent replication of this phenomenon occurred in June 2022 when a Google AI engineer declared the LLM he was training had become sentient (). Such attributions will likely increase as newer LLMs demonstrate increasing proficiency in seemingly distinct human qualities, like humor (, 39).

The means by which LLMs process, interpret, and generate information is a highly technical field requiring specialization in natural language processing, statistics, computational linguistics, and machine learning. As a historian, I lack the technical knowledge to evaluate the merits of these debates. However, I can assess the GPT series in terms of their historical accuracy. From my observations, LLMs offer significant opportunities to for digital historians, but also cause for concern.

Judge for yourself in the following example. In the code below, GPT-4 is prompted to generate responses to the introduction of the "Philosophy of History" entry in the Stanford Encyclopedia of Philosophy by Daniel Little (). GPT-4's responses do not rely on any training or previous input from the user, nor does using GPT-4 require any specialized programming knowledge or hardware – only an OpenAI account and an API key or direct access via the ChatGPT browser.

While acknowledging the caveats noted earlier, GPT-4's responses to these prompts reveal an impressive range of its abilities to both "interpret" language and generate plausible responses to instructions. Prompt 1's text summary is concise, accurate, and accessible. Prompt 2 provides an accurate translation of the first sentence of Little's introduction. Prompt 3 presents a haiku close to a 5-7-5 structure that aptly captures the spirit of the main text. Prompt 4 correctly associates idealism with Hegel, the figure linked to the philosophical approach of Absolute Idealism ().

Prompt 5 is revealing in other respects, particularly concerning the shortcomings of these models. The citations seemingly offer appropriate works on the philosophy of history, and GPT-4 excels in even identifying bibliographic details. However, most LLMs are quite prone to generating citations where the author, publisher, and publication date cannot be confirmed with reference databases like WorldCat – in other words, they are invented by the model.

These inaccuracies are a phenomenon described by AI researchers as "hallucinations." Such hallucinations represent a major challenge in LLM research and for practical applications of this technology. () Indeed, the confident assertions of such factual inaccuracies pose a real danger in many domains, particularly given the remarkable effectiveness of these models in generating convincing and otherwise accurate prose. Detecting such errors can be difficult, as initial testing by OpenAI on the GPT series demonstrated that human readers often struggle to identify text generated by LLMs ( 16, table 7.3). Rectifying such hallucinations is a significant area of research. However, some scholars, like computational linguist Emily Bender, argue that such behaviors are inherent flaws in LLMs (). Given this tendency, historians must exercise great caution when employing this technology.

Per Bender: “GPT-3 can’t ‘compose nonfiction’. Nonfiction by definition is factual writing about the world. But GPT-3 has no access to facts, only to strings in its training data. To the extent that it outputs strings of words that humans interpret and can verify as factual, that factuality is and can only ever be purely accidental.”

The capabilities of LLMs, along with their flaws, stem from the vast dataset used to train them: the Internet itself. The data collection built for training GPT-3 encompassed the majority of English-language Wikipedia, Reddit's thousands of discussion forums, extensive corpora of digitized books, and a filtered (yet immense) collection of billions of web pages contained in the Common Crawl repository (, 8-9). While OpenAI's training process aimed to remove potentially offensive texts, the sheer scale of the dataset made selective curation impossible. Consequently, LLMs generate responses reflecting both the best and the worst of our online world.

This reality has troubled previous AI implementations. Well-intentioned researchers have created chatbots that spew hateful invective, human resources applications that refuse to hire female applicants, and algorithms based on criminal justice sentencting guidelines that starkly reinforce racial disparities already prevalent in the carceral system (). The GPT series has been known to unexpectedly generate responses in innocuous contexts containing violent imagery, sexually explicit language, and racial, ethnic, and religious slurs (). These findings further confirm the prescient warnings offered by scholars such as Safiya Umoja Noble (), Timnit Gebru (), Ruha Benjamin (), Kate Crawford (), and Trevor Paglen () on digital practices that reinforce analog inequalities. Some AI researchers consider such behaviors as lamentable but solvable problems through further technical advances. Reducing the impact of biases is a significant research area, particularly through the creation of smaller, more carefully curated datasets for AI training. However, many historians will likely share the skepticism of some researchers concerning such mitigations. () Bias emerges from more than just explicit language or imagery but also from the very structures of societies. Can any historical source be separated from its context as a neutral artifact, free of its creator's perspective and the influences of its time? What about the untold millions of sources that make up the scale of an LLM's training set?

Historians should contribute to the broader dialogue about the implications of these technologies, especially as they become increasingly embedded in our digital lives. Yet, such flaws do not mean LLMs have no place in the historian's toolkit. In fact, by acknowledging and confronting these shortcomings, historians can better contribute our disciplinary perspective on the debates concerning this technology – particularly in leveraging the strengths of these models to empower and broaden accessibility. The case studies below demonstrate how foundational AI models can serve as a versatile tool for both researching and communicating the past.

Case Study: Oral History Transcriptions¶

A promising approach for LLMs and other foundational AI models is for data preparation and cleanup. A general rule of thumb is that 80% of the labor involved in data analysis is dedicated to preparing the data (, ix)). AI models hold significant potential to streamline and accelerate the challenging work of creating "tidy datasets" ().

Oral history provides a particularly useful case study for demonstrating the potential value of AI models. Transcription of audio recordings is a central activity in this methodology, but transcriptions often require considerable expense, labor, and time. However, advances in machine learning models have resulted in impressive gains in streamlining this task. Notable among these models is Whisper, an open-source audio transcription model developed by OpenAI that belongs to the same Transformer family as the GPT series (). Let's test Whisper on the first two minutes of a transcribed oral history of historian John Hope Franklin by the Southern Oral History Program ().

OpenAI released Whisper as a series of open-source machine learning models freely avaliable and hosted on HuggingFace. () However, for simplicity this demonstration code uses OpenAI's API for Whisper. As of March 2023, OpenAI charged $0.36 per hour of recorded time for transcriptions using the API.

Whisper provides state-of-the-art transcriptions, but as of March 2023 doesn't perform speaker diarization. To faciltiate text comparasion, I've added diarization to Whisper's transcription in the 'diarized_whisper_transcript' variable. A variety of tutorials have been published for implementing speaker diarization, timestamping, and other features for Whisper. ()

Several notable takeaways can be observed in comparing the original transcript with Whisper's transcription. Whisper misses Franklin's interjection in section 1, omits part of the final sentence in section 11, and misidentifes "Nashville" for "international" in section 13. It also offers some variations in puncuation for sections 6, 7, and 10, and transcribes common filler words usually omitted.

Human review is still required to ensure a faithful transcription. Yet those reviewers will start with a remarkably accurate transcript produced quickly and at little cost. Applications like Whisper hold significant potential for oral historians, as they can dramatically improve the efficiency and cost-effectiveness of transcription workflows. As this technology continues to advance, it is likely that the accuracy of tools like Whisper will only improve, further enhancing their utility for oral historians and allowing them to focus more on the analysis and interpretation of these sources.

Case Study: Error Correction of Optical Character Recognition Scans¶

Another potential use case of AI models for digital historians is the error correction of optical character recognition (OCR) scans. Machine learning techniques, such as those pioneered by the research team at Transkribus, have greatly enhanced the quality, speed, and cost of OCR scans across a broad range of historical texts. () However, even high-fidelity OCRs possess error rates that have insidious impacts on the accessibility and searchability of texts collections. () For instance, the image below comes from a newspaper published in a German prisoner-of-war camp in Mississippi during World War II and later microfilmed by the Library of Congress. Let's compare an OCR scan of this image via Google's Cloud Vision OCR service with a human transcription of the same text.

While the image quality is satisfactory and the text is printed using modern typefaces, the OCR still generates errors requiring human correction. Correcting such errors necessitates substantial review and intervention, representing a significant labor for processing a sizable text corpus. However, LLMs can expedite this correction process when guided by a carefully designed prompt. This practice, often referred to as "prompt engineering," is a method used to direct LLMs in completing specific tasks. Details concerning each prompt and the methods behind it are described in the hermeneutical layer.

Let's now see how GPT-4 interprets this prompt, and compare the generated corrections with the initial OCR scan.

Prompt engineering utilizes the emergent capabilities of LLMs to solve complex tasks. This prompt employs two common prompt methods:

Few-shot learning: In this prompt, two sample OCR generations (Sample Text 1 and Sample Text 2) are provided, along with their corresponding corrected transcriptions (Corrected Transcription 1 and Corrected Transcription 2). These examples serve as a learning basis for the model to emulate the task and perform it on a new OCR generation. This technique allows the AI to generalize from the provided examples and apply its insights to new instances. ()

Chain-of-thought reasoning: The prompt guides the model through a series of steps to complete the task. It starts with examining the sample texts and their corrections, noting the formatting in the corrected transcriptions, and then proceeds to work on a new OCR generation. It is theorized that this structured approach helps LLMs better determine the desired output, which in this case is a more accurate transcription. ()

The versatility of these prompting methods enables the completion of a broad range of tasks, as does the ability to prompt LLMs using natural language instructions. Understanding and developing prompt approaches is an active area of research and experimentation. An excellent resource for exploring prompt engineering has been published by DAIR.AI (Democratizing Artificial Intelligence Research, Education, and Technologies). (

Prompt 1: OCR Correction

You are an AI research assistant with a specialty in correcting errors in OCR scans of newspaper images. In the following task you will be given a OCR’d text, and you will generate corrections using the following Task Format. Follow the instructions of the Format step-by-step.

Task Format

1. Examine the Examples: Examine the two examples of sample OCR generations, Sample Text 1 and Sample Text 2.

2. Examine the Corrections: Examine the two examples of OCR corrections given, Corrected Transcription 1 and Corrected Transcription

3. Compare the changes between the Samples to the Corrected Transcriptions. Note Formatting in the Corrected Transcriptions: Within each corrected transcription are symbols to represent uncertainty or substantial edits to the original OCR. These are inserted to communicate to the user which words may need additional human review. Words that you are very uncertain about are bracketed with a ***\ before and after the very uncertain word.

4. Examine New OCR Generation: You will then be given a new OCR generation.

5. Generate New Corrected Transcription: Based on the examples and the prompt instructions, compose a New Corrected Transcription based on the New OCR Generation. Do your best to make it as accurate as possible. Do not correct the grammar or wording of the text, only seek to correct errors in the OCR. Likewise do not add umlauts, eszetts, or other diacritics.

Let’s begin.

Sample Text 1

#begin sample text 1

EIN LLERRUNDGANG

it diosom Boitreg wird der Boricht des Oborfoldwebels ltstaodt uobor seinen Rund- gang durchs Lagor c.bieschlosson. Im Juli bozogon dig orston PoWs ihra “zwungsheimat” im Camp I. wio sah es desnals in diesom Lagor aus ? Bcreckon vurde:: aufgebaut;

#end sample text 1

Here is Sample Correction 1.

#begin sample correction 1

EIN LAGERRUNDGANG

Mit diesem Beitrag wird der Bericht des Oberfeldwebels Altstaedt ueber seinen Rundgang durchs Lager abgeschlossen. Im Juli bezogen die ersten PoWs ihre “Zwangsheimat” im Camp I. Wie sah es damals in diesem Lager aus? Barecken wurden aufgebaut

#end sample corrrection 1

Here is Sample Text 2:

#begin sample text 2

Dem crsten bericut, der den kunagang duro: das dritte bateillon schilcerte, lesson wir heuto den ueber das zwcito bataillon folgen. Die Schriftl.

im bingang zum zweiten Lataillon ruht der blick auf dor Lagerstresse, did scharf ansteigt Lirks und rechts zichon sich sauber geinauerte Graeben entlane

#end sample text 2

Here is Sample Correction 2:

#begin sample correction 2

Dem ersten Bericht, der den Aufmarsch des dritten Bataillons schilderte, lessen wir heute der Ueber das zweite Bataillon. Die Schriftl.

Am Anfang des zweiten Bataillons ruht der Blick auf der Lagerstrasse, die scharf ansteigt. Links und rechts sich sauber eingegrabene Graeben entlang.

#end sample correction 2

Here is the New OCR Generation:

#begin new corrected transcription without umlauts, eszetts, or other diacritics

In comparing the two outputs against the human transcription, GPT-4 demonstrates a remarkable ability to correct the errors in the OCR scan. GPT-4's ability to indicate its uncertanity in its error corrections also speeds human review of its output. However, the quality of the initial OCR scan remains crucial for a successful output. For example, here is an lower-quality image containing 'noise' that causes substantial errors in the OCR output.

Here we see that GPT-4 achieved only modest improvements in the OCR output, perhaps an indication of the limits of this approach. However, GPT-4's multimodal nature may open new opportunities for the future. GPT-4's multi-modal interface will soon allow it to directly perform prompted OCR scans of images. () Only time will tell if these abilities surpass existing OCR techniques. Yet there seems to be remarkable potential for further exploration.

These two case studies demonstrate an LLMs capacity to assist in various forms of data cleanup and preparation. While human review remains essential, LLMs can make that review less time-consuming and labor-intensive. LLMs are already being employed for tasks as varied as text normalization, metadata generation, automated summarization, date extraction and standardization, sentiment analysis, relationship extraction, and named entity recognition. Further experimentation will undoubtedly reveal future use cases. Such approaches can improve the accuracy, lower the costs, and accelerate the pace of data preparation. LLMs can also expand accessibility to historical sources, enabling the use of programmatic techniques via natural language instructions.

Case Study: Ask-A-Source - Retrieval Based Methods for LLMs¶

While LLMs demonstrate a broad range of capabilities for data cleanup, their tendency towards 'hallucinations' represents a formidable obstacle towards their use in historical research and analysis. However, recent advances in retrieval-based methods offer potential to ground LLMs in greater factual accuracy. () Such techniques also enable the use of LLMs to analyze large text collections, search the Internet, and utilize external tools to solve problems in unfamiliar knowledge domains. The following case study demonstrates one such approach for historians: how an LLM can be used to answer questions about a historical source.

One of the shortcomings of LLMs is their context length, or the hard limits on how much text they can interpret in a single query. Models like GPT-3 and ChatGPT can only process textual data for some two to three pages in length, while GPT-4 possesses a much larger context length. () Yet even the most advanced models cannot directly interpret long-form texts or large text collections in a single query.

Yet these limits can be circumvented through the use of tools like semantic search and prompt chaining. In this case study, two different AI models will work together to answer questions about a historical text, Thomas More's History of Richard III (). At the end of the process, GPT-4 will deliver a series of responses supported by direct quotations from the text.

Tne first step in this process is semantic search with OpenAI's ADA model, a computational technique for establishing text similarity. Let's pose the following question: "Who killed the princes in the Tower?"

Here are the sections of the text identified by Ada as the most semantically similar:

To enable semantic search in More's text, it must be converted to text embeddings, an approach for transforming "unstructured text data into a structured form." (("Text Embeddings Visually Explained" 2022)). Preparing a text for semantic search depends on the intended use case and requires consideration of the model's context length. In this example, the text was broken down to the paragraph level and accompanied by a summary initially generated by ChatGPT (GPT-3.5) and edited by the author afterward. OpenAI's Ada model was then used to compute a set of searchable embeddings. Both the original data file and computed embeddings are included in the Github repo for this article. See the OpenAI Cookbook for step-by-step code examples of how to generate embeddings for texts. ((OpenAI Cookbook 2023)) A variety of other semantic search platforms are available, such as Pinecone, Weaviate, and Haystack.

This student edition of More's History of Richard III is produced by the Thomas More Society. () My thanks to Dr. Ian Crowe, director of the Thomas More Program at Belmont Abbey College, for the opporunity to explore this text with his students in spring 2023 with this research approach.

The results from the semantic search provide the three sections of the text with the highest semantic similarity score. However, high semantic similarity does not always indicate relevance, and such searches can return false positives. Filtering these false positives is essential if you wish to attempt large-scale analysis of an entire text. We can use an LLM to determine the relevance of each text section and then filter out irrelevant sections before answering the question.

In making multiple queries to the LLM for a single task, we will use a technique known as prompt chaining. Prompt chains break down complex tasks into smaller, more manageable components by making a series of queries to the LLM. We'll use the langchain library for creating these chains. ()

The first link in the chain is a relevance check. In this sequence, GPT-4 will be prompted with a detailed set of instructions, along with three examples of how to determine relevance. Then each text section identified in the semantic search will be passed to the LLM. GPT-4 will use the prompt to generate an analysis of each section's relevance.

langchain is a Python library for large language model programming. It is supported by a highly active open source community, and supported by extensive documentation.

This prompt design employs few-shot learning and chain-of-thought prompting to guide the language model in determining the relevance of a given text section to a specific question. Let's analyze each aspect of the prompt design and its impact on the model's response.

Chain-of-thought prompting: The prompt design incorporates a set of instructions for step-by-step completion of the task. This method guides the model through the process of analyzing the text section and evaluating its relevance to the deaths of the Princes.

Few-shot learning: As in the previous example, a set of examples are offered to help guide the model's response. Here various sections of the text are compared with a user question for relevance. A scripted sequence is then provided, following the chain-of-thought instructions. Both irrevelant and revelant examples are used.

Prompt 2: Textual Relevance

You are an AI expert on the ‘History of Richard III’ by Thomas More. In this exercise you are given a user supplied question, a Section of the Text, a Semantic Similarity Score, and a Method for determining the Section’s relevance to the Question. Your objective is to determine whether that Section of the text is directly and specifically relevant to the user question. You will be the Method below to fulfill this objective, taking each step by step.

Here is your Method.

Method: Go step by step in answering the question.

1. Question: You will be provided with a user question.

2. Section: You will be given a section of the text from Thomas More’s ‘The History of Richard III.’

3. Semantic Similarity Score: You are then given a semantic similarity score, which ranges from 1.0 (highest) to 0.0 (lowest). The higher the score, the more likely its potential relevance. Scores approaching .90 and above are generally considered to have some relevance. However, this score isn’t fully determinative, as other semantically related words in the Section can generate false positives. Weigh the value of this score alongside a careful examination of the Question and the Section.

4. Key Words: Identify key words in the Section that are specifically and directly related to the Question. Such key words could include specific locations, events, or people mentioned in the Section.

5. Background knowledge and context: Use your background knowledge of the subject matter to further elaborate on whether the Section is directly and specifically relevant to answering the Question.

6. Relevance Determination: Based on your review of the earlier steps in the Method, determine whether the section is relevant, and gauge your confidence (high, medium, low, or none) in this determination. High determination is specifically and directly related to the Question. If the section is relevant and ranked high, write ‘‘Section_x: Relevant’. Otherwise, if the section is not relevant and the determination is less than high, write ‘Section_x: Irrelevant’.

7. Relevance Explanation: Based on your review in the earlier steps in the Method, explain why the Section’s relevance to the Question. Let’s begin.

Prompt 2: Examples for in-context learning.

Example 1:

1. Question: Who is Cecily?

2. Section:

Summary: Section_113: In a sermon at Paul's Cross, it was revealed to the people that King Edward IV's marriage was not lawful, and that his children were bastards.

Text: Section_113: Now then as I began to show you, it was by the Protector and his council concluded that this Doctor Shaa should in a sermon at Paul's Cross signify to the people that neither King Edward himself nor the Duke of Clarence were lawfully begotten, nor were the very children of the Duke of York, but gotten unlawfully by other persons by the adultery of the Duchess, their mother, and that also Dame Elizabeth Lucy was verily the wife of King Edward, and so the Prince and all his children were bastards that were gotten upon the Queen.

3. SSS: 0.869 Based on the provided information, it appears that the section is potentially relevant to the question. The semantic similarity score is relatively high, indicating that there may be some connection between the section and the question. However, it is important to carefully examine the section and the question to determine the specific relevance.

4. Key Words: The key words in the section that may be specifically and directly related to the question are 'King Edward,' 'Duke of Clarence,' 'Duke of York,' 'Elizabeth Lucy,' 'Prince,' and 'children.' These words refer to individuals or groups of people mentioned in the section.

5. Background knowledge and context: Knowing that the question is asking about a person named Cecily, we can use our background knowledge about the context of the text to further assess the relevance of the section. The section mentions several individuals and groups of people, including King Edward, the Duke of Clarence, the Duke of York, Elizabeth Lucy, the Prince, and the children. Cecily is not mentioned by name in the section.

6. Relevance Determination: Based on the key words identified in the section and our background knowledge of the context, it is unlikely that the section is relevant to the question. The section does not mention the name Cecily and does not provide any information about her. Therefore, I have a low degree of confidence in determining that the section is relevant to the question.

7. Relevance Explanation: The section is not relevant to the question because it does not mention the name Cecily and does not provide any information about her.

8. Final Output: Section_113: Irrelevant.

Excellent. Let's try another.

Example 2:

Edward Question: "What was King Edward IV's appearance?"

Section 3: Summary

King Edward IV was a good-looking and strong man who was wise in counsel and just in war. He was also known for his love of women and good food. However, he was also known to be a fair and merciful man, and he was greatly loved by his people.

Section 3: Text

King Edward IV was described as a goodly personage and very princely to behold. He was courageous at heart, politically astute, unshaken in adversity, and joyful in prosperity without being overly proud. In peace, he was just and merciful, while in war, he was sharp and fierce. He was bold and hardy in the field, but he didn't take unnecessary risks. Observers of his wars would commend his wisdom when he withdrew, just as they would praise his manhood when he emerged victorious.

Edward had a lovely visage, and his body was mighty, strong, and well-built. However, in his later years, he became somewhat corpulent and burly due to overindulgence in food, but he still maintained an attractive appearance. In his youth, he was greatly given to fleshly wantonness, a fault that the people did not hold against him since it didn't involve violence and affected only a few. Furthermore, this fault diminished and was eventually abandoned in his later years.

Semantic Similarity Score (SSS): 0.928

To determine the relevance of this section to the question, we can follow the steps of the Method:

1. Question: The user's question is "What was King Edward IV's appearance?"
2. Section: The given section is about King Edward IV's appearance, character, and behavior.
3. SSS: The semantic similarity score (SSS) is 0.928, which is above the threshold of 0.90, indicating potential relevance between the section and the question.
4. Key Words: Key words in the section that are directly and specifically related to the question include "goodly personage," "visage lovely," "body mighty, strong, and clean made," and "somewhat corpulent and burly." These words directly describe King Edward IV's appearance.
5. Background Knowledge: Based on background knowledge of the subject matter, it can be confirmed that this section is directly and specifically relevant to answering the question about King Edward IV's appearance.
6. Relevance Determination: The relevance determination is high, as the section is directly and specifically related to the question.
7. Relevance Explanation: The relevance explanation is that the section contains detailed descriptions of King Edward IV's appearance, including his physical appearance and any changes to it over time.

Final Output: Therefore, the final output is "Section_3: Relevant."

Excellent. Let's try another.